Repository navigation
fix(expander): do not copy a template node's GPU usage to the node it simulates - #842
Merged
Merged
Conversation
… simulates To find a node type that fits a pending Pod, the expander copies an existing node and its GPUs and checks whether the Pod fits the copy. The GPU copies kept the template's usedBy and phase. When the template's GPUs were held by native Pods (usedBy=nvidia-device-plugin under progressive migration) or were not Running, PhaseFilter rejected every copy, and the expander reported "failed to satisfy the pending pod, no potential GPU nodes can fit" without creating a NodeClaim. The copy stands for a new, idle node: set usedBy to tensor-fusion, phase to Running and clear runningApps, as Available is already reset.
|
🎉 This PR is included in version 1.65.17 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
To find a node type that fits a pending Pod, the expander copies an existing node and its GPUs (
prepareNewNodesForScheduleAttempt) and checks whether the Pod fits the copy before it creates a NodeClaim. OnlyAvailablewas reset on the GPU copies: they kept the template'susedByandphase.Under progressive migration the hypervisor marks GPUs that native Pods hold with
usedBy=nvidia-device-plugin. When every candidate template node's GPUs are held by native Pods or are notRunning,PhaseFilterrejects every copy. The expander then reportsfailed to satisfy the pending pod, no potential GPU nodes can fitand creates no NodeClaim. In a production cluster that mixes native and TensorFusion Pods on the same nodes, Pods stayed Pending with exactly this event. The controller logs showed the cloned GPUs dropped atPhaseFilter.Fix
The copy stands for a new, idle node. Set
usedBytotensor-fusionandphasetoRunning, and clearrunningAppson each GPU copy.Tests
TestPrepareNewNodesForScheduleAttemptResetsTemplateGPUState: template GPUs held by a native device plugin andPending, with progressive migration on. Every copy must passPhaseFilter, and the template GPUs must be left untouched. Without the fix,PhaseFilterkeeps 0 of 2 copies.make testpasses.make lintreports 4unparamfindings in existing test files;origin/v1reports the same 4 without this change.The same fix for main is in #
.